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ABSTRACT 

Of concern in the problem-based learning process in 
medical student education is the effect of allowing students to 
generate their own learning issues while discussing cases and 
problems. This study considered learning issues in problem-based 
learning, studying faculty and student overlap and consistency across 
learning groups while adding the perspective of analyzing learning 
issues developed by students without the assistance of a tutor. Lists 
of students* learning issues, interviews with the course director and 
course writer, and final examination scores were collected for 4 of 
12 tutorial groups, representing 52 students, and 4 cases prepared by 
students. The mean case overlap in issues and content covered was 
somewhat lower for this study than for previous studies using staff 
tutors. Not having a tutor present during the initial case discussion 
may slightly reduce the amount of case overlap between 
faculty-intended learning objectives and the learning issues 
generated by students for the purposes of self-study. Only 
one-quarter of the 77 faculty objectives were identified by all 4 
groups as learning issues. Results suggest that reliance on a 
tutorless format may not be appropriate when other sources of 
structure are absent from the curriculum. (Contains three tables and 
seven references.) (SLD) 
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Objectives 

Of concern in the problem-based learning (PBL) process in medical student education is 
the effect of allowing students to generate their own learning issues while discussing cases and 
problems. At UCLA we have used problem-based learning groups without faculty members 
present when beginning discussion of a new case. What types of learning issues do students 
generate when left without faculty guidance? How do the student-generated items compare to 
faculty objectives under these conditions? Is there a relationship between the number of student- 
generated learning issues and performance on the final examination? 
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Perspective: Literature Peview 



There have been few studies of the nature and role of student-generated learning issues in 
PBL. Most of these studies have examined the overlap between student-generated learning 

issues and faculty-identified objectives. Coulson and Osborne ^ examined the relationship 
between faculty objectives for a patient problem and student-identified learning issues. They 
found that, on average, 5 student groups were sufficient to discover all faculty-identified learning 

issues. Dolmans, Gijselaers, Schmidt and Van der Meer^ studied the process of learning issue 
identification, and distinguished Type A mismatches, a faculty objective not identified by a 
group, and Type B mismatches, issues identified by a group that were not faculty objectives. 
They found that the average degree of overlap between faculty objectives and student-identified 
issues in a small-group session was 64%. 

Two additional studies examined student-generated learning issues in relationship to self- 



study and other curricular features. Dolmans, Schmidt and Gijselaers^ studied the relationship 
between student-generated learning issues, faculty objectives and self-directed learning 
activities. They found that group-identified learning issues were not, apparently, the sole factor 

influencing self-study. Dolmans and Schmidt^ then used a questionnaire to discover what course 
aspects influenced students’ learning activities during self-study. They found that the 
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importance of group learning issues as a starting point for individual study increases during the 
four years of medical school. Students’ learning activities during self-study are also influenced 
by the reading material suggested for the course, course objectives (i.e., as stated in the course 
book) lectures, self-assessment tests in course books, other students, and the tutor. 

One study examined the range and consistency of learning issues across multiple groups 

discussing the same case. Kennedy and Wilkerson^ had students in problem-based learning 
groups in an anatomy/ histology course individually list the major issues studied as a result of 
discussing cases with 3 different types of pathologies. They identified four types of learning 
issues: structural, functional, clinical and psychosocial. The consistency among groups was 
highest for gross anatomical issues. 

The present study connects these two approaches by considering faculty and student 
overlap and consistency across groups while adding a new perspective — an analysis of learning 
issues developed by students without the assistance of a tutor. 

Methods 

We used a case study approach^ to explore the relationship between student-generated 
learning issues, faculty learning objectives, and examination performance in tutorless problem- 
based tutorial groups in a gross anatomy course for first-year medical students. The course was 
part of the first semester of medical school so students had no prior exposure to problem-based 
learning. Students were randomly assigned to tutorial groups. We selected 4 of the 12 tutorial 
groups, representing 52 students in the course, as a sample of convenience. The groups met 
weekly throughout the semester and considered each individual case for two sessions. The initial 
session, on all except the first case, was conducted by the students without a faculty member 
(tutor) present, and without a designated student leader. Multiple lectures and gross anatomy 
dissections were also a part of the course. 

Three types of data were collected -- lists of students’ learning issues, interviews with the 
course director and case writer, and final examination scores. During the process of beginning a 
new case, students listed their self-generated learning issues on large sheets of newsprint. Since 
no faculty tutors were present during these sessions, these issues represented students’ perceived 
needs for review of prior knowledge, reinforcement of content presented in lectures and 
laboratories in the same course, or acquisition of new knowledge. The sheets were retained by a 
group member and used during the second tutorial session for each problem to structure the 
discussion and to update the faculty tutor on the work done since the first tutorial. 

Sufficient documentation was available to study four of the five cases which were begun 
without a faculty tutor present. Case 2 was a 42-year old male who came to the emergency 
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room after an episode of chest pain while exercising. Case 3 was a 48 year-old man with liver 
failure after a long history of alcoholism, who was being considered for transplantation. Case 4 
was a 29-year old woman in whom a left adenexal mass was discovered during her fifth week of 
pregnancy; she feared a serious birth defect in her unborn child. Case 6 was an 18-year old male 
who nearly drowned during a surfing accident which left him a paraplegic. The newsprint sheets 
for cases 2, 4, and 6 were available from three groups, and case 4 for all four groups. We 
initially reviewed the sheets for all cases and groups to determine common themes. Five themes 
appeared to characterize the issues: 

• normal structure and function 

• injury and disease states 

• diagnostic procedures 

• treatments 

• psychosocial issues 

We then sorted the student'-’ learning issues into these categories. 

Next, one of the authors (JLED) interviewed the course director and the case writer. 

They had selected each case to stimulate study of a particular organ system but had not 
developed specific learning objectives since they felt that, taken with the accompanying lectures 
and laboratories, the intended learning objectives would be fairly obvious. During the 
interviews the course director helped develop a master list of primary and secondary learning 
objectives for each case. In addition, he reviewed the categorized list of students’ learning 
issues and added objectives to the master list when he felt that students had identified an 
important issue that he had overlooked. 

Third, we identified the final examination scores for the 52 students in the four groups. 
The examination consisted of multiple choice, extended matching and slide or tissue 
identification items. A mean examination score was calculated for each group and group means 
were compared using multiple t-tests. 
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The course director identified a total of 77 learning objectives for the four cases. Table I 
illustrates the breakdown of these objectives by priority and theme. 

Table I 

Number of Faculty Objectives for Each Case by Priority and Theme 
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Table II shows the number and percent of faculty learning objectives identified by 
students for each case in each group. We have labeled this case overlap (issues identified by a 
group, when groups are considered one at a time). 



Table II 
Case Overlap: 

Number and Percent of Faculty Objectives Identified for Each Case by Each Group 
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Table III shows the percent of mean case overlap (mean percent of identified objectives 
for a case, averaged across all groups) compared to potential overlap, the percent of faculty 
objectives identified by at least one group (identified by any of the groups, when considered all 
at once as one large group). It also indicates Core overlap, the percent of faculty objectives in 
each case that were identified bv every group studied. 

Table III 

Mean Case Overlap, Potential Overlap, and Core Overlap: 

Percentage of Faculty Objectives Identified by Students for Self-Study 
During a Tutorless Problem-Based Learning Discussion 
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In this study case overlap ranged from 45% for the surfer case (Case 6) to 67% for the 
chest pain case (Case 2), resulting in a mean case overlap of 54% for students working without a 
faculty member (tutor) present. This is somewhat less than the comparable value of 64% found 
by Dolmans, Gijselaers, Schmidt, and Van der Meer^ for groups with tutors present. 

Overlap can also be considered as the percent of objectives for each case which were 
identified by at least one group. This form of overlap, which we refer to as potential overlap, is 
useful in determining if a case is constructed in such a way as to make the objectives readily 
accessible to students. Viewed in this way, potential overlap ranged from 76% for the surfer 
case (Case 6) to 100% for the chest pain case (Case 2), as shown in Table III. Students were 
slightly more likely to identify objectives from the categories of treatment (92%), injury/disease 
(88%), and diagnostics (88%) than they were for normal structure-function (82%) or 
psychosocial issues (75%). Ninety percent of the primary objectives and 80% of the secondary 
objectives were identified by at least one group. Only 1 1 of the 77 faculty objectives (14%) were 
not identified by any student group. This compares favorably with Dolmans who found that 15% 
of faculty objectives were not identified in her .study. 



' case overlap: identified by a group, when groups are considered one at a time . 

’ potential overlap: the percent of faculty objectives identified by at least one group (identified by any of the 
groups, when considered all at once as one large group). 

’ core overlap is the percent of faculty objectives in each case that were identified by every group studied. 



Finally, overlap can be considered as those faculty objectives identified by all groups 
(core overlap). Such a measure might be helpful in identifying the core content of a course 
composed of multiple groups working on the same problems. Core overlap ranged from 12% for 
the surfer case (Case 6) to 38% for the chest pain case (Case 2). Interestingly, Case 1, in which a 
faculty tutor was present in each group during both sessions (to introduce the technique) had a 
core overlap of 53%. One quarter of the 77 faculty objectives were identified by all student 
groups as learning issues across all four cases considered. This percentage was much higher in 
Dolmans at 41%. 

The mean examination score for each of the four groups ranged from 86 to 89. Multiple 
t-tests demonstrated no significant difference in scores among the four groups. 

Discussion 

In problem-based learning, the faculty prepares problems or cases for discussion by 
multiple small groups. Each case is selected and composed to stimulate the discussion and study 
of key learning objectives. The degree to which students working in a collaborative small group 
setting will identify learning issues for self-directed study that correspond to the faculty’s 
objectives can be used to tell us something about the work being done by individual groups, 
variability among multiple groups as they discuss the same case, the power of a case to stimulate 
specific learning issues, and the core content covered by every group. Of particular interest in 
the present study was the lack of a faculty tutor to direct the students’ discussion as they 
developed their learning issues during the initial session on each case. 

A view by cases 

The concept of case overlap is a useful one for examining the performance of individual 
cases and of individual groups. If the mean case overlap value is 54%, we might want to look at 
those cases in which the percentage is much lower; here the surfer case. Case 6. Is there some 
reason that students might not be identifying learning issues that are concurrent with the faculty 
objectives in this case? No group identified topics related to cervical function, injury to cervical 
ligaments, electrolyte imbalance, or psychological problems associated with sudden paraplegia. 
For example, it is not surprising that students overlooked the electrolyte issue which was slipped 
into a dependent clause of a single sentence, e.g., “After correction of his electrolyte 
abnormalities due to near salt-water drowning, and stabilization of his pulmonary function, Mr. 
Johnson is transferred to Rancho Los Amigos Hospital for surgical stabilization of his cervical 
fracture.’’ 
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A view by groups 

We can also look at the average case overlap for each group. A consistently low case 
overlap percentage in any one group might suggest some problem with the way that the group is 
identifying its learning issues. For example, one of the groups that we studied tended to identify 
fewer numbers of faculty objectives. The faculty tutor, when asked, indicated that the students 
were not consistent in writing down their identified learning issues. 

A view of the course 

Finally, case overlap can be averaged across all cases to tell us something about the 
course. What percentage of faculty objectives are being identified by students as learning 
issues? Students in our four tutorless groups identified an average of 54% of the faculty’s 
learning objectives for the four cases examined. Dolmans et al’ reported an average case overlap 
of 64% in groups with tutors. The comparison suggests that tutors may be helpful in prompting 
students to identify learning issues that correspond to the intent of the faculty. It is also possible 
that the type of case used in the two different schools, and the number of possible faculty 
objectives developed for each case, play equally important roles in increasing or decreasing the 
match between students’ learning issues and faculty objectives. The mean number of faculty 
objectives per case was 4.3 in the Dolmans study and 19 in our study. The cases used in the 
Dolmans study were simple physiologic states in need of explanation. One example given was 
only one paragraph in length while our cases were complex, data-rich patient cases two to three 
pages in length. 

Potential overlap indicates how well each case serves as a stimulus for the faculty 
objectives. In the four cases that we studied, 66 of the 77 faculty objectives were identified by 
one or more student groups. We called this feature potential overlap. It suggests that the cases 
were appropriately structured to raise the intended objectives. We are in the process of 
examining the 1 1 objectives that were not identified, in the light of case details, to determine 
what changes might need to be made in the case presentation or in the objectives themselves. 
This is particularly important since we cannot rely. on faculty tutors (who are not present) to 
provide guidance. One strategy that we have used is the inclusion of two to three suggested 
study questions at the end of Part One of each case. In a subsequent observational study of the 
tutorless groups, we found that some student groups paid little attention to these questions when 
determining learning issues. 

One of the concerns that many faculty members express about the student-directed nature 
of problem-based learning is the lack of a common curriculum across student groups. This 
concern seems justified by the results of our study which demonstrated that only one quarter of 
the 77 faculty objectives were identified by all four groups as learning issues. With a large 
number of groups participating, however, this core overlap might be more usefully defined as 
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issues identified by 90%, (or a similar figure) of the groups, since variability among groups as 
they pursue different sections of a case makes 100% identification difficult when 10 to 15 groups 
are considered. The chest pain case (Case 2) stimulated the most concurrence with 38% of the 
faculty objectives identified by all four groups. This case also had the highest case overlap, the 
highest potential overlap, and the fewest faculty objectives. The high overlap scores could be a 
reflection of the fact that, of the four cases, the one involving chest pain is a more familiar 
scenario to anyone who reads widely or encounters medical drama in the popular media. Also 
the fewer faculty objectives increases the likelihood of student concurrence. Given the lack of 
difference in the examination scores among the groups, it may be that as Dolmans and ■Schmidt'* 
found, students study topics derived not only from the cases, but from other aspects of the course 
as well, such as lectures, laboratory exercises, the text, and the course syllabus. This broad study 
agenda may serve to offset the lack of a larger common core in our problem-based learning 
curriculum. The mean examination scores of the groups, ranging between 86 and 89, suggests 
that students are learning a majority of the material in the course even if the small group 
discussions are not uniform in their coverage of content. 

In conclusion, the mean case overlap was somewhat lower in this study than that reported 
in two previous overlap studies'"’ using staff tutors. Not having a staff tutor present during initial 
case discussion may slightly reduce the amount of case overlap between faculty intended 
learning objectives and the learning issues generated by students for the purposes of self-study. 
However, it is essential to note that problem-based learning in this course is accompanied by 
lectures and laboratories from which students can draw additional clues about what topics are 
essential to study. In a recent study of the relationship between exam performance, tutor 
expertise, students’ prior knowledge, and course structure, Schmidt^ noted that “students need a 
minimum level of structure in order to profit from problem-based instruction.” He concluded 
that this structure can be derived by students on the basis of their prior knowledge, the structure 
of the curriculum, or the expertise of the staff tutor. In our anatomy course, the structure 
provided by laboratories and lectures may have eliminated the need for faculty tutors. The 
tutorless format was initiated at UCLA to reduce the demand on faculty time, involving faculty 
tutors when their expertise is most crucial — during the return to a case as students teach one 
another what they have learned during self-study. The reliance on a tutorless tutorial format 
may not be appropriate when other sources of structure are absent from the curriculum. 
Comments on further work 

During the Fall semester of the school year 1994-1995 three of the four cases were 
repeated, with the lowest ranked case on all measures, the surfer case. Case 6, being replaced. 
The newsprint sheets were collected from a total of 1 1 groups. The data so far appear quite 
similar to the present study, providing more robust numbers, and are still being analyzed. 
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Additionally, one author. (JLED) sat in on the problem-based learning sessions for three 
of the eleven groups during two cases. Verbatim audiotapes were made and transcribed, and she 
annotated, with a second recorder, the behavioural aspects of the sessions; seating arrangements, 
the roles students performed, and the ways they interacted. 

Some of the results of these studies include: 

• Since the original faculty objectives were stipulated after the cases had been studied 
by the 1993-1994 students, the present study focused on Dolmans et aP Type A 
mismatches: a faculty objective not identified by a group. Dolmans Type B 
mismatches: issues identified by a group that were not faculty objectives, were 
virtually nonexistent since the student lists of learning issues were used to assist in 
reviewing and adding to the initial faculty objectives. The 1994-1995 data include a 
number of Type B mismatches which are being inspected and categorized. 

• A decision has been made to post the faculty’s suggested objectives after the second 
session of each case, to allow groups to measure their success at achieving faculty 
goals and reevaluate their methods. The continuing feedback in both directions will 
allow for further refinement of the case as well as of student skills. 

• Discussion is now occurring regarding the installation of one or more sessions at the 
beginning of the school year to train student participants in the more metacognitive, 
process-oriented aspects of group work, to heighten their awareness of group 
function, to improve communications, and to strengthen their ability both to work 
cooperatively and to keep the problem-based learning goals in mind. 

More work needs to be done looking at the ways students interact in the groups, 
particularly with regard to the teaching of the learning issues to each other. Even if a group has 
correctly identified all faculty objectives and individually learned their separate issues, if they are 
unable to share the information in an appropriate manner to assist other group members some of 
the purposes of problem-based learning are being subverted. Some of this could also be 
addressed by point 3, above, training students in the praxis of group process. 
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